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Agenda 


•  Defining  characteristics  of  current  large  product 
development  projects 

•  Technical  demands  on  project  management  and  limitations 
of  most-used  project  management  techniques 

•  How  dynamic  modeling  has  helped  The  Aerospace 
Corporation 
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Defining  characteristics  of  large  product 
development  projects 


•  Structural  complexity 

-  Multiplicity  of  elements  (organizations,  resources,  tasks,  etc.) 

-  Various  types  of  interdependency 

•  Pooled  (resources) 

•  Sequential  (tasks) 

•  Reciprocal  (feedback) 

•  Uncertainty 

-  Goal  (requirements) 

-  Methods  (processes) 

•  Tight  time-constraint 

-  Underestimation 

-  Political  factors 
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State  of  the  Practice  in  Project  Management 


•  Existing 

-  Models  activities  and  dependencies 

•  PERT  charts 

•  Gantt  charts 

-  Resource  leveling 

•  Project  management  software, 
e.g.,  Microsoft  Project 


-  Earned  value  management 
•  Lagging  indicators  of  progress 


CPl/SPl 


C  PI2  50  4  50  4.57  4  66  4  68  4  71  4 J24.73  4.73  4.74  4.74  4.74 
SPI1!  .45  5-66  4.09  4-33  4-1 0  4.23  4.10  3.93  3.82  3.84  3.86  4  02 
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Challenges  to  Project  Management  Current  Practice 


•  Feedback 


CTest 
Fix 


•  Downstream  effects  of  quality  problems 


•  Effects  of  waiting  for  work  products  and 
resources 


•  Intangible  factors 

-  Schedule  pressure 

-  Morale 

-  Overtime  effects 
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How  dynamic  modeling  has  helped  The  Aerospace 
Corporation 

•  Program  office  support 

-  Lessons  learned  with  Dynamic  COQUALMO 

-  Test  and  fix  modeling 

•  Acquisition  planning 

-  Modeling  probabilities  of  successful  completion 

•  Research 

-  Study  of  concurrent  processes 
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Lessons  Learned  with  Dynamic  COQUALMO 
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COQUALMO 


Defect 

Introduction 


Code  Defects— ► 


Residual  Defects 


Design  Defects^ 


Requirements 

Defects 


Defect  Removal 

Peer  Reviews,  Automated  Analysis,  Execution  Testing  and  Tools 

Extension  of  COCOMO  II 

-  Relates  defectivity  to  cost  and  schedule 

-  COCOMO  II  drivers  are  treated  as  quality  drivers 

-  Quality  measured  in  counts  of  non-trivial  defects  (critical  system 
function  impairment  or  worse) 

Submodels 

_  OefeCf  introduction  COCOMO  H  and  COQUALMO  were  developed  at 

the  Center  for  Systems  and  Software  Engineering 

_  UQfQQf  removal  of the  University  of  Southem  Ca|if°rnia- 
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Defect  Introduction  Submodel 


Sources  of  defects:  Requirements,  Design,  and  Code 

21 


D1 source  =  DIRsource,nom  *  Size8™™  *  ]~~[  Deject  Driver, 


source 


i-\ 

Dl  =  defects  introduced  from  each  source 
DIRnom  =  nominal  defect  introduction  rate  by  source 
Size6  =  software  size  raised  to  scale  factor  by  source 
Defect  Drivers  in  Quality  Adjustment  Factors  (QAFs) 

-  Example:  Analyst  Capability  (ACAP) 

Defect  driver  values  produced  through  a  two-round  Delphi  process. 


ACAP  Level 

Requirements 

Design 

Coding 

Very  High 

.75 

.83 

.90 

High 

.87 

.91 

.95 

Nominal 

1.0 

1.0 

1.0 

Low 

1.15 

1.10 

1.05 

Very  Low 

1.33 

1.22 

1.11 
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Defect  Removal  Submodel 


•  Defect  removal  activities:  peer  reviews,  automated  analysis,  testing 

3 

DR  artifact  ~  DI  arfijaCf  *  J” 1 0  DRF^ar^j-act) 

i- 1 

•  DR  =  defects  removed  from  artifact 

•  Dl  =  defects  introduced  into  each  artifact 

•  DRF  =  removal  fraction  for  each  activity,  /,  applied  to  each  artifact 

•  DRF  assigned  to  quality  levels  of  activities  in  2-round  Delphi 
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Model  Description:  Defect  Flows 

•  Three  inflows,  one  each  for  requirements,  design,  code 

•  Outflow  for  each  review  type,  automated  analysis,  and  testing  phase 

•  Flows  arrayed  by  interval 
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Major  Defect  Discovery  Profiles  for  Projects  A  &  C, 
actual  and  modeled 


0  20  40  60  80  100 

Project  Time  (month) 
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Number  of  Defects  Avoided 


Study  of  Concurrent  Processes 


PCON,  TOOL,  DOCU 
&  RESL 

•  PCON,  DOCU  &  RESL 

PCON,  TOOL  & 

*  DOCU 

PCON&DOCU 

•  RESL 

•  PMAT 

PCON 

•  TEAM 

•TOOL 

DOCU 

1 

- 1 - 1 - 1 - 1 - 1 

Estimated  Additional  Cost  to  Project 
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Cumulative  Defect  Count 


Study  of  Concurrent  Processes 


0  20  40  60  80  100 

Project  Time  (month) 
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Duration  of  Test  and  Fix  Cycling 
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The  difficulty  of  estimating  test  duration 


•  A  discovery  process  versus  an  insurance  process 

< - > 

A  terrifying  adventure  Validating  what 

into  the  unknown  we  already  know 

•  Some  factors  in  test  duration 

-  Amount  of  quality-inducing  effort  applied  prior  to  testing 

-  Type  and  complexity  of  software  (including  architecture) 

-  Organizational  knowledge  of  the  product 

-  Organizational  discipline  (change  mgmt,  SCM,  build  planning ,  etc.) 

-  Types  of  testing  required  (high  reliability  requirements?) 

-  Resource  constraints  (people/facilities) 

-  Product  (software,  test  cases,  test  tools)  availability 

-  Duration  of  individual  activities 

>  Many  factors  contribute  to  software  readiness 
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Previous  Approaches  to  the  Question 


•  Linear  estimates 

Number  of  tests  X  Average  time  for  each  test  =  Total  test  duration 

•  Expert  judgment  plus  Monte  Carlo  sampling 


Optimistic  Estimate  of  most  Pessimistic 

estimate  likely  duration  estimate 


•  Software  project  estimation  tools 

Development  Time  =  2.5  (Effort  Applied)b  [months] 


Software  project  type 

b 

Basic 

0.38 

Intermediate 

0.35 

Highly-constrained 

0.32 

>  Each  of  these  methods  has  peculiar  limitations  _ 

»  0  AEROSPACE 


An  example  test-and-fix  (TaF)  duration  model 


Set  TF1  Availability 
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Discovering  significant  factors 


•  Used  a  full  factorial  experiment 

-  Use  constant  inputs  representing  expected  operational  values 

-  All  combinations  of  four  factors  at  two  levels  each  (24):  16  simulation  runs 

-  Response  variable  is  duration  of  TaF  process 


Factor 

Low  Value 

High  Value 

Test  Facility  Availability 

60  hrs /  week 

100  hrs/week 

Runs  per  Test  Case 

2 

8 

Test  Run  Duration 

2  hrs 

5  hrs 

Fix  Time 

24  hrs 

96  hrs 

•  Analysis  of  variance 

-  Calculate  percentage  contribution  to  variation  in  duration  from  sums  of 
squares 
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Significant  factors 

60%  n 


50% 

40% 

30% 

20% 

10% 

0% 


Runs  per  Test  Case  and  Test  Run  Duration 
interact  to  produce  an  effect  in  addition 
to  their  individual  effects 


A:  Runs  per  Test  Case 
B:  Test  Run  Duration 
C:  Fix  Time 

D:  Test  Facility  Availability 


1 


^  ^  O  &  4> 


° 
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Discovering  behavior 


•  Used  an  additional  full  factorial  experiment  to  produce  response 
surfaces 

-  Focus  on  Runs  per  Test  Case  and  Test  Run  Duration 

-  Use  one  Fix  Time  value  and  two  Test  Facility  Availability  values 


Factor 

Values 

Test  Facility  Availability 

60  and  100  hrs/week 

Runs  per  Test  Case 

2,  4,  6,  8 

Test  Run  Duration 

2,  3,  4,  5  hrs 

Fix  Time 

7  days 

21 


0 AEROSPACE 


Test-and-Fix  Duration  (weeks) 


Behavior:  the  TF  threshold 


Test  Facilities  Availability:  100  hrs/week 


Factor  interaction  above  the 
TF  full  utilization  threshold 


50-60 

40-50 

30-40 

20-30 


TF  availability  moves  the 
threshold 


Test  Facility  Availability:  60  hrs/week 


I  50-60 
I  40-50 
I  30-40 
I  20-30 
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Modeling  a  likely  scenario  and  alternatives 

•  Used  likely  inputs  to  estimate  the  duration  of  the  test-and-fix  cycle 


Factor 

Values 

Sample 

Test  Facility 
Availability 

Both  test  facilities  at 

40  hrs/week  each 

Constant  for  all 
simulation  runs 

Runs  per 
Test  Case 

(2,  .1),  (3,  .1),  (4,  .3)  (5,  .2), 

(6,  .1)  (7,  .05),  (8, .05) 

Randomly  for  each  test  case 
in  each  simulation  run 

Test  Run 
Duration 

Triangular(2,  3.5,  5)  hrs 

Randomly  for  each  test  case 
in  each  simulation  run 

Fix  Time 

(7,  .125),  (8,  .125),  (9,  .125), 

(10,  .125),  (11,  .125),  (12,  .125), 
(13,  .125),  (14,  .125)  days 

Randomly  for  each  test  cycle 
of  each  test  case  in  each 
simulation  run 

•  Alternative  scenarios 

-  Additional  test  facility  availability  or  an  additional  test  facility 

-  More  optimistic  Test  Run  Duration  and/or  Fix  Time 
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Test  case  completion  times 
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Findings  and  impacts 


•  Good,  early  estimate  of  test  duration 

-  Eliminates  re-planning  activities 

•  Identify  the  primary  factor  in  test  duration 

-  Focus  on  the  real  problem 

-  Avoid  expensive,  inconclusive  experimentation  on  the  actual  system 

•  Understand  system  behavior 

-  Identify  test  management  options 

•  Staffing  levels 

•  Test  facility  availability 

•  Degree  of  overlap  in  test  and  reviews 

•  Rate  of  taking  test  cases  into  TaF  cycle 

•  Order  of  test  cases 

•  Backlogging  defects 


>  Actions  supported  by  analytic  underpinnings  of  statistical  modeling  _ 

2=  0  AEROSPACE 


Acquisition  Planning 
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Acquisition  Seat  Creation  in  the  Concept  Design  Center 

Process  and  Tool(s)  Repeat  for  Each  Concept/Configuration 


Reference 

Documents 


1. 

DoD  5000.2 

2. 

JCIDS  3170 

3. 

FARs  > 

4. 

DFARs 

5. 

Defense  Acq 

Guide 

Choices 

Topics 

Primary 

Program  Type 

A  CAT  1 

1.  Changes  to  basic  Acquisition  doctrines  and  processes 

a.  Standard  (large)  Air  Force  program 
procurements 

Standard 

b.  Standard  procurements  with  ability  to  insert 
waivers 

c.  New  and  special  Acquisition  doctrines 

i.  "faster,  better, 

cheaper" 

Preliminary 
Acquisition 
Strategy  Panel 
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Concurrent  Decision  Support  at  Aerospace 

Concept  Design  Center  (CDC)  &  Concurrent  Program  Definition 
Environment  (CPDE)* 


CDC 


Customer  Inputs 

Performance 

Requirements 

Optical  Aperture 
Data  Collection  Rates 
Onboard  Processing 
Power  Modes 
Data  Volume 
Data  Latency 
Data  Downlink  Rates 
Comm  Bandwidth 
Comm  Power 
Pointing  Accuracy 
Pointing  Knowledge 

Mission  Requirements 

Constellation  Size 
Orbit(s) 

Mission  Lifetime 

SV  Design  Life 

Survivability 

Radiation 

Maneuvers 

Availability 

Replenishment 

Design  Constraints 
Tech  Freeze 
Launch  Vehicle 


Ground 


Software 


Payload 


C&DH 


Comrnm&C 


AD  ACS 


Propulsion 


Thermal 


Power 


Structures 


Life  Cycle  Costs 
(Dev,  Prod,  O&M) 


Subsystem  Mass  &  Power  Estimates,  SVMass  &  Power  Contingencies,  SVDiy  Mass,  Mission 
delta-V,  PropellantMass,  EOL  Power,  BOL  Power(SolarArray  &  Batter/  sizing), Thermal 
Cycling.Technology  Readiness,  Design  Heritage,  CommAvailabilities&Durations,  Inertias, 
_ Deployed  &  Stowed  Packaging,  LV  Margin,  SLOC,  Facilities,  Personnel _ 


<: 


Feedback  from  Acquisition  to  Design 


cfdc 

Acquisition  Strategy 

Blocks,  Spirals 

Contracting 

Down-Selection 

CONOPb 
Operational  Views 

Performance 

Capacity 

Quality 

Utility 

Availability 

Schedule 

Milestones 

Tech  On-  &  Off-Ramps 

Software 

Management  Strategy 

Funding 

Budget  Profile 

Risk 

Identification 

Mitigation 


Program  Mgmt. 
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Acquisition  Enterprise  Model 

Upper  left  hand  portion  of  Model  shown  in  detail 


1  ExtendSim 

File  Edit  Text  Library  Model  Database  Develop  Run  Window  Help 

[o*  &  »  ioq'%  jei  ♦  to  >ir©  t  ^  ^mugcr- 

i  ® 

-h  Enterprise  Acquisition  Process  Model  1101 19.mox  <C:VDocuments\Aerospace\Acquisition  Planning: 


Rejected  outright^ 
End  after  waiting  period  ■= 
MDAP  threshold  crossed  = 
End  @  MAJCOM  A= 

Kill  by  MDA  at  Concept  Dedsioi 
Kill  at  AoA  Check  = 
End  Process  at  COA= 
Kill  at  MS  A  Decision  = 

Kill  at  MS  B  Decision  = 

Kill  at  MS  C  Decision  = 
Program  Kill  atPDR  = 
Program  Kill  atCDR  = 

End  at  MS  C  Decision  ■= 

End  Death  at  AFROC  joint  inter  preA= 
EndDeath  at  AFROC  joint  integ  preA  = 
End  Death  at  AFROC  indep  Pre  A  = 
End  Death  at  AFROC  joint  interest  pre 
End  Death  at  AFROC  joint  integ  pi  ' 
End  Death  at  AFROC  indep  Pre  B 
End  Death  at  AFROC  joint  intpreC  = 
EndDeath  atAFROC  joint  integ  preC  = 
End  Death  atAFROC  indep  Pre  C  = 
End  prior  to  start  of  Requirments  Swimlane  PreB  = 
End  prior  to  start  of  Requirements  Swimlane  PreC  = 


Qi— I 

a/  OS  for  First  Run  onfy 


ILL 


Pre  MS  -  A 

Pre  MS  -  B 

Pre  MS  -  C 

JCIDS 

Part 

ppb&ep  Shown 

DAS,  Govt 

DAS,  Contractor 

new  lane  TBD 
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#  of  Programs 


Histograms  of  Formal  Process  Time  to  MS  C  by  ACAT 


□  ACAT  I  no  excursions  ■  ACAT  II  no  excursions  □  ACAT  III  no  excursions 
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Study  of  Concurrent  Processes 
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Phase  Relationships  Example 


Leading  increment 


Trailing  increment 


*  Work  available  and  inherited  defects 
*-  Rework  found  in  dependent  phase 
-*■  Staff  allocation 
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Rework  Implications  of  Concurrent  Engineering 


f 


Phase  1 

Phase  2 

Phase  3 

Phase  4 

Phase  1 

Phase  2 

\hase  3 

Phase  4 

>  Potential  regression  at  the  “ILA”  anchor  of  the  Trailing  Increment 
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Duration 


>  Duration  risk  increases  with  degree  of  concurrency 
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Duration  in  the  Better  Quality  Scenario 

Less  defect  introduction,  better  discovery 
Duration  reduced  ~17  months 
No  duration  benefit  to  concurrency 
Duration  risk  substantially  reduced 


Duration  in  the  Larger  Staff  Scenario 

Double  staffing 

Duration  reduced  Vs  to  30  months 
Duration  risk  dramatically  reduced 
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Conclusion 
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Experiential  Lessons  from  Dynamic  Modeling 

•  Dynamic  modeling  is  useful,  often  necessary,  for 

-  Gaining  insight  into  the  nonlinear  processes  of  programs 

-  Estimating  outcomes 

•  We  are  learning  to  use  it  in  project  management 

-  Asking  the  right  question 

-  When  the  cost  is  justified 

-At  this  point,  limited  engagements  work  best 

-  Can  be  used  to  identify  dominant  process  constraints 

•  Valuable  tool  for  research 
-Across  projects 

-  Process  concepts 
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